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Background 

The study discussed here used a "text sefnantic" approach to investigate 
the processes readers use to construct an understanding 'of test items selec- 
ted from among the norm referenced standardized reading comprehension tests 

i : 

most frequently used by schools across the United States (note 1).. The-broad 

• •*«.. 

goal^of our project were 1) to identify aspects of; test language and struc- 
ture that might interfere with .%emprehensio^, 2) to. describe the skills and 
knowledge necessao(/or a reader to comprehend each passage, and 3} to • 
examine the extent to which young readers have that knowledge and those 
"Skills. Every aspect of our project — the formulation of our research' 
questions, our, system of- text analysis, and our interpretation of the data — 
was deeply influenced by our view of reading comprehension as a dynamically 
constructive process (Anderson, 1977; Bartlett, 1982; Goodman & Goodman, - 
1978;- Polanyi, 1966; Schank & Abel son, 1977; Rumelhart, 1977). 

We conducted our, inquiry in two phases.,. The first phase focussed on- 
the development of'a systematic procedure for jthe text semantic analysis of 
-test items. Qur analysis aodifies.the lexical, syntactic, and rhetorical 
structures occurring in a specific text as well- as -the linguistic and con- 
ceptual' demands these structures presumably. pose for a reader '(Fillmore, 
1981b). it alsoVacks. the manner in which the genre, the content, and the ' 
Linguistic material of the text sha^pe the developing meaning. . 

,»his t system of text analysis served as the basis for the next phase of 
the st'udyVan investigation of readers' comprehension' and question-answering 
strategies. \Using a variety of interview procedures and metacognitive probes., 
we gathered 'data which permitted us to analyze aspects of test items (each 
consisting ow 



a paragraph and accompanyins multiple-choice questions-) that 



posed difficulties for actual readers. We compared readers as'meaning- 
integraters with readers as question-answerers in order to determine whether 
the test items we examined identified as high scorers the same students we 
identified as successful meaning-makers. « • 

The project itself can^e characterized as^a series of descriptive : 
studies carried out^in the tradition of linguistic inquiry whereby our ob- 
•servations of readers' performance were used to inform pur. developing ana4 

lyses of 'texts and readers, and these in turn were validated using -other 

/: <•-■•■'' 
test items and other readers. All our generalizations, therefore, are an 

»• > » " ' 

outgrowth of these repeated, in 'depth observations. 
* ' » 

Ideal and Actual.. Readers * * ' 

The analysis of a particular text proceeds linearly, tracing the changing 
interpretations that an "ideal reader" cou\dV)USti.fiably make in the process 
of comprehending the text. -The system conveys a dynamic and idealized 
version of reader-text interaction in„that it codifies- a finite number of 
cognitive ^expectations and integrations warranted by a particular text; these 
interpretations constitute a comprehensively fcWIoping env.isionment of 'text 
meaning up to "any specific point within the^xi. 

The ideal reader, then,- i? a$ abstraction pf that knowledge and those 
skills required for a particular interpretation of a particular text. The' 
ideal reader is equipped with the^array 4 of schematic know! edge. that the text . 
presupposes for its interpretation,,, arib Tacks 4 (but is^repa^e^to learn) the ' 
material that the.te^t introduces — material that -the reader is- not vet ' 
presumed to know. The kinds of schemata'o'r knowledge structures utilized. 

• ' » * - * 

by the ideal gsaderfor aqy. particular text are' evoked by T) the" concepts ' 
or lideas expressed, 2) the overall.! -structure; and organization which may , ■ 



rely on such devices as conversational inference, indirect speech acts, ' 

« 

and a story grammar, and 3) the internal language (the grammar in its - 
broadest sense, including the lexicon^ Our x analysis of the'envisionmervt 

** « , ' 

constructed by the ideal reader portrays N the manner injvhich these knowledge 
structures operate and interact/in the developing meaning -of a pass6ge % 
This notion of ideal reader is not to be confused with a "mature" ' 
or. "real" reader.. Every .reading experience, even among fully competent 
readersj is somewhat* idiosyncratic, especially since good readers may use 
-somewhat different processing operations to arrive at a range of acceptable 
interpretations. Real life reading' experiences 'also differ from our ".ideal" 
in that reader^ have varying ways of interacting with text depending on"' their 
purpose for reading. Tor example, an actual' reader does not necessarily 
want to make a complete interpretation, when quick'ly reading a "whodunnit" 
mystery, but most- certainly does want to make a fuller interpretation when 
preparing for an essay examination. Our analysis does notVcoupt for such 
alternative purposes for reading (although theoretically it could). 'The 
ideal re'ader we have created' for this project? is -a very specific idealized ' ' 
reader designed to exemplify the kinds of cognitive operations deemed ap- 
propriate and useful in processing specific texts. (For a more complete ex-^ 
amination of the ideal reader concept, .see Fillmore, 1981a and Kay, 3981.) 

( Our procedures for text\analysis focus on the Ijexical, syntactic, and ', • 
rhetorical structures that occur in consecutive meaning segments, the struc r ■ 
tures that contribute to the develdpiogi^'e-nv^sionment'', of the meaning of. > 
'tnat .particular passage.. Codification permits us to locate the schemata 

and the schematic links an ideal reader draws upon. or needs to. construct fn 

... « • . 

order to envision what is happening at any point in the text. Through'this 



> 

procedure we are also able to see where ..these processes are interrupted 
or thwarted by tlje text. When the ideal reader is left with unanswered.- 
questions, our theory judges the text to be defective — in some way in- 
coherent t>r inadvertently misleading. In turn, we can use , our analysis 
to compare hoW rea\ readers depart from the ideal reader in developing an 
envisionment of the meaning of a particular passage; we can identify gaps 
in a reader's world knowledge or strategic knowledge ~ gaps, that have in- 
inhibited comprehension of the text. 

The System of Analysis 4^ * ■ ■ 

• As \ passage is processed schemata are evoked anp then* integrated, as 
the processing continues. ^Meaning derived from any given portion of the 
text is shaped fay hbw earlier segments were interpreted' and continues to ' * 
develop and change in light of letter segments. ■ These .changing "envisionments 
are a^ record the "text internal" world that is constructed by the reader 
while prpcessing the text. The^e enyisionments are the primary ."dynamics" 
through which the reader experiences the "message." Our system of analysis 

keeps track of the envisionments throughout the reading experience. It a!so. 

* * * * 

-kee k ps track of the cognitive pVocesses executed iji the reader's effort to 

construct these envisionments.. \ . ' • 

. Since any text includes some concepts that are* explicitly stated and 

W e * . 

many others that require varying degrees of inference, .we distinguish among 
a variety of levelV of envisionment that correspond to the degree of text- 

.based or reader*-based influence. AltHotigh greater oY fewer distinctions "are : 

* ' * • " • i - •* 

possible in categorizing* levels of envisiosment, we have most\rec£ntly been • 

working with- four levels: s ^ " . ' . . \- * 



0 - The most literal level of the envisionment, includingv only those 
elements that 'are directly warranted by what' the text says; 

, E 1 - That. level which, requires text-based inferences to be made; 
E 2 - That level which -requires inferences triggered- by the text,, but 
which is leased on personal knowledge or assumptions; ' 

E 3 " That. level 4 of the reader's "text-world 1 : that is not warranted by 
the text but represents the idiosyncratic embellishments of " an actual 
reader, (E 3 by definition tioes ,'not occur in our analysis of "the ideal 
readeV.)- % • 

Our analysis of a text keeps track o'f .the changing levels of envisionment 
constructed by the ideal reader. It also specifies the schemata an ideal 
reader*ffright use to construct, .at a particular point in time',-an acceptable 
envisionment. We can also follow the envisionments by actual readers to - 
compare these with ideal readers,' constructions." • 

Domains of analysis . For our analysis* we have distinguished three 
sources which simultaneously influence the developing* envisionment: genre, 
content, and '.text. 1 ' N • - . 

Genre (Gn) concerns itself with readers' hypotheses. about the kind of 
text they are reading. A genre hypothesis is generally made/>arly in. the 
reading experience (as early as the first sentence) and influehces a reader' 
expectations about what the text will "say" and how it wiVI "end." ' •' ' 
* - Content (Co) refers to the" evolving base of information and events. 

T f xt (Tx) refers to the grammatical and Rhetorical aspects of the text; 
tfiese features create relationships that help readers raise questions .and 
resolve hypotheses. ..." • . . 



, . In responding to elements, in these three domains, readers perform 
a number of different types of cognitive operations* in the" process of gen- 
erating meaning/, Our 'system of analysis tracks six general operations: 

1. Questions (Q) — uncertainties the reader has at any point 
during reading. < « ' * * 

>2. Hypotheses (HVJ — predictions fhe reader makes about what the . 
genre is, about what the function of a particular piece of text is, or about 
the answer to a question* based on a specific portion of the* text. 

(H-*) predictions, the reader makes about what will be "said" 
in succeeding portions of the text. 

3. Assumptions (Ass) « mearvings the reader takes for granted without 
textual evidence. 

4. Schemata, (Sch) basic memory structures evoked about genre^rcon- 

I i \ 

tent', or text. f - 

** * 

5. Conclusion (Con) — information which substantiates a hypothesis. 

6. Validation (Val) ~ proof that a hypothesis was correct or a 

*" * ' 't 

(H-y) fulfilled. • / , 

' / . * ' 

Additional processes wepe identified in some cff our notations artd our system 

was refined as the Rroject progressed. (For a detailed presentation of our 

system of -analysis and notation see Fillmore and Kay', 1930 and 1982.) 

« 

Figure 1 conta-ins an abbreviated example of the opening segment of one 
test. passage annotated using our system (note 2). The left column presents 



. Insert figure 1 about here 
» ------ . # 

our ideal*. reader analysis of the text segment, tracing -in detail the sources 
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■ ? - v BRONCO BUSTER 

(1) If a bronco /buster wants to win a rodeo contest . 

L0(1) If-clause, present tense, indefinite articl es 



. text Analysis 

1. Sch (Co) CONTEST 

PARTICIPANTS, DESIRE TO WIN, COMPETITION, 
RULES, CRITERIA FOR- DECIDING ON WINNERS.., 

2.. Sch (Co) RODEO -'' ■ 

CONTESTANTS, 'AUDIENCE, EVENTS, RIDING, 
ROPING, TYING ... 

3. SCHEMATIC LINKS 

a RODEO presents a number of CONTESTS 

4. Sch (Tx) N+N COMPOUND ? 
N identifies a type of N 

5. Sch (Cof BRONCO BUSTING 

HORSE: WHEN WILD WILL TRY TO THROW RIDER 
RIDER: RIDES WITHOUT GETTING THROWN, BY 
- BREAKING, WILL OF HORSE RENDERS HORSE 
TRAINABLE 

6. SCHEMATIC LINKS 

BRONCO BUSTING is -one of • the kinds of 
-events in a RODEO 

7. Sch (Tx) N+V-er COMPOUND • 
compound designates someone who 
V's and'N's 

8. Sch (Tx) CONDITIONAL SENTENCE . 
ANTECEDENT, CONSEQUENT 

9. Hyp^ (Gn) GENRE-EXPOS PROSE ; 



10. Hypl (Th) THEME=WINNING RODEO CONTEST 



11. Hyp->(Tx) 

the next clause will be a<modal clause, 
expressing obligation . - , 

12.. Hyp J, (Tx) * 

the two noun phrases "a bronco buster" 
and "a rodeo contest" are to be 
interpreted as generic 

13. Ass (Co) £} 
• when bronco busters choose* to enW rodeos, 
they So so because they want to win 



Related Interview Questions 

1. What do you think .happens in 
a contest? 



What kinds of things happen 
in a rodeo? 



4. What do you think they meant 
' ^ by "rodeo contest?" 

5. What is a bronco buster? 
What does a bronco buster do? 




Figure 1 

8 



| 7« ("What does somebody who is a 
pig-washer do? What about a 
-maijbox painter?") * 



9. What kind of passage do yon '* 
0 think this is?' Do you thinL-.it 
mfght be the kind of thing you 
might read in a story book?;. etc. . . 

10. What do you think this passage 
•is gping to be about? 

11. Wftat do- you think the next 
couple *of y/otds are going to 
be? 

\ 

12; From what We know so far, is 
this, story going to-be about 

k how to be a bronco, buster or 
about one bronco buster in 
particular? 

13. If bronco busters decide to be 
*in a rodeo contest do you think 
. they want to -Gin, or doesnVjt it 
make a difference?^ 



v • . V 

• ^ of the developing envisionments.. In turn, this*detail allowed Cte to develop 
a comprehensive set of questions, to probe the developing environments of . 
real readers confronted with these" texts. The questions 'generated for this 
particular segment are listediin the' right hand columnjn figure'l/ 

/ * 

' », Procedures . 

. / • • • ' 

Subjects .J 

During the two years of Jhis study, many texts were ^selected from an 
array of standardized tests and analyzed using the procedures described , ' 
above. Readers of a variety of ages were then interviewed following the 
.general format described below. Over the course of the study, a continuing ' 
.pattern of te"xt analysis followed by observation of real readers was used 

.* *■ • 

to refine both the text analysis system and the interview procedures.' This 

. effort also provided us with a wide range of examples of how envisionments 

'• ' . ■• * 

develop (or go astray) across a text; these shaped our interpretation of 

the specific set of data reported below, and will be drawn upon in some 

» detail in the general disxussion of results. ' i 

The major analyses. to be reported here are based on intensive study 
V ' « 

of the developing environments of 26 third graders in response to the ' - 

test item "Bronco Buster" (figure 2). The student's were. selected from a 

middle class elementary school in Oakland, California and have a varied 

ethnic and home language background. Because our focus was on comprehension 

.... 4 * "» 

of test items, students with decoding problems were not included in- our 

< ■ ' ■ ' 

sample. Results of the Califortjja Test of Basic Skills, Grade 3, were * 

available for all students. Their percentile mean for the vocabulary* sub- 

. Insert figure 2 'about here '* 

• * . ■ 

ERJC • / y ' lo . . , 



. * {JL_a ^ronco buster wants to win a rodeo contest, he 
must observe the contest rules. One of these rules is that 
the rider must keep one hand in the air. A rider who does 
not do this is disqualified. 

1. A bronco buster who ignores the rubles is 

A. skillful S B. disqualified C. chosen D. winner 

4 * 

' 1 

2. lira rodeo contest a bronco'buster must keep one hand 
A. under B. still ^ free D. hold 



Figure 2: Bronco Buster 
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test was 70.55 their comprehension mean was 73.3.. 

Interviews . * 

The .text was presented to students, in meaning segments corresponding 
to those used in describing the developing envisionment.^f.'the ideal reader.' 
After a student had read a segment, quesl|qns such as- those' in the right 
hand column of, figure 1 were asked. Because readers derive some meaning 
from clues in the prior text, and from such features' as sentence length, 
paragr^h length » and text length, each segment appeared' with all previously 
read segments showing, and unread .segments indicated bu^concealed. When 

-J ' 

reading the second segment qf Bronco Buster, for example, the students were 
presented witff the item illustrated in fipure 3. This procedure necessitated 



» •• Insert figure 3 about here 

that each short text be presented in a packet of several pages before it 
could be completely revealed. ... ■ % 

Since text segmentation of this type may introduces variables that differ 
from those a 4 student mi$ht ervcoanter when reading the same text in an un- 
mutilated version, two different interview procedures were 'developed. Half ' 
the students began by reading an* unmutilated versipn of the text, followed 



.by an oral retelling. They were'tjren presented with the mutilated version' - 
and its accompanying questions, segment-by-segment. „ The ofher half of the 
students began with ,tfie segmented text and accompanying questions, without 
the opportunity 'to read akd- retell the text as a whole. Interviews for 

•7 9 

bothgroups began with a training session to familiarize the students with • 
.the' mutilated text format, and ended with a general question about the "mairL 
point" of the story. Resul/ts f ro m the two procedures were essentially 



12 



identical and will be discussed together in the sections below.'-' 
t Each interview .session lasted between 45 minutes and one hour. 'Each 

session was taped and transcribed. The transcripts were coded in terms ■ 
of specific aspects of the reader's developing envisionment, and evalua- 

-» • 

V 

ted^to determine each student's overall ability to construct a reasonable, 

• , o , 

evolving envisionment of the meaning of the'text. These analyses pter- 

. mitted us to make comparisons across groups of children as well-as to- 

describe in depth the performance of individuals. ' ./ 

<* « 

Results and Discussion. 
The. data will be, reported and discussed under two general headings: 
1) observations about test takers; and 2) observations about tests. - A 
final section wi 11 'h^cjhlight-some problems to consider wh?n writings 
taking tests, or using their results. 
/Responses to Questions » 

* This section will contain a review of the Bronco Buster questipns^nd 
responses; what answers the students expected, what answers they chose-, ^ 
and why (see figure 2 for the complete test item). After reading^each 
passage segment-by-segment, students were aske^Lto read and answer the 
first question, without being shown the multiple-choice selections. They 
were then shown the choices ^ and^asked to make a selection. Also, they 
were requested to explain wfiy they "liked" the word they selected and did " 
not "like" the words they did not select. (For an analysis of howtest 
questions relate (or do not relate) to £he, passage, see Coleman, 1982.) " 

Responses to the tw°o Bronco Buster questions present ah interesting 
contrast: 20 of the 26 students eventually selected the correct answer 

->6^om among the four alternatives in each item, but the answers they, 
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expected before "seeing the choices were quite different. 
Question 1 ; A bronco buster who ignores the rules is 

< 

A. skillful- 'B. disqualified C. chosen D. winner 
- • . C 
For the first question, even before seeing the choices, 14 students * 

(54%) anticipated that "disqualified" would be .the correct response, • 
eight were unable "to answer, ahd four gave other responses. All 14 who"- 
. anticipated "disqualified" also .selected it as the correct response, as 
did another six students who had not originally anticipated that response. 
Of the six who selected incorrect responses to this item, four chose ^ 
"winner" and two chose "skillful." . * * \ . 

Because our interview' questions are keyed to important aspects^of the 

V 

reader's developing envisionment, we can locate quite precisely the ' 
comprehension processes that led some readers to select the -wrong responses. 
In this case, most of the problems stemmed from laclTof knowledge of two 

' words: ^disqualified* (which appears in this passage and in the question)" 
and "ignores" (which appears" poly in the question).' Although, the (a t 1 , 
children who anticipated /'disqualified" were familiar with its meaning, 
those who selected iffrom among the other choices had difficulty pronouncing 
it, were uncertain pf its meaning, and selected it either because ft seemed' 
to be a key word in the passage or- simply due to the visual match from 
pas-sage to response. Of the four students whd Those "winner," two ex- 
plained that if you do- not follow the rules you" 11 cheat and thereby increase 
your likelihood of winning..^ Although they- did not know what "disqualified' 1 
meant, a combination of context clues and world knowledge led them to con- 

'struct a reasonable rationale for "winner" as the correct' response, the 



f 

. remaining four students were '.not familiar with either "ignore" or "dis- 

.qualified" and chose "skillful" or "winner" based on their knowledge of 
, a bronco busteV.s prowess. For at least those four readers, uncertainty 

about the meaning of the question was an~obVious impediment to their 

selection of the correct response. 
. If the test-maker intended the question to measure students' ability 

to gain the meaning of a difficult word .("disqualified") from the context " 

of the passage, the inclusion of another difficult word ("ignores") in tfie 

question thwarted this goal. « 

Question 2: In a rodeo co/itest a bronco buster must keep one hand 

A. under , t B. still C. free D. hold - ' 
After reading the stem to the second question, 17 s.tudents (65%) anticipated 

that "in the air" would be the response, -a plausible conclusion to the stem 
although .it does not appear as one of the choices. Five students who 
anticipated "in the air" 'got this question wrong when .forced to select from 
the alternatives given; four chose "hold" and one chose "still." (The 
other incorrect response was also "hold" -- by a student.who did not offer 
any anticipated response.) * , 

For this test item, the students' explanations about the plausibility , 
of each response are especially revealing. Although finally rejecting it, 
two children explained that "under" was a logical response, reasoning that 
the bronco buster has to keep^one hand in the air and the other hand under 
the rope. .Four children considered selecting the word "still .'^reasoning 
that if, you do not hold your hand jtill you will o fail off. One child 
stated that if you hold one hand in the air the other one has to be stiff. 



Eight students considered the word "hold". at some point because if you don't 

hold you'll fall off. Two youngsters reasoned that if you keep one hand in 
-t » 
. the air you'll have to hold with the other one. The 26 readers- did not all 

•' :have a complete envisionment of .the passage. Some did not know what a 
bronco buster was (interpreting bronco as a model namewfor an automobile), 
some confused the person with the animal, and still others were^uncertain- 
as to -the kind of animal (cow,- bull) a bronco was." However, all youngsters 
demonstrated some understanding that a bronco buster keeps one hand in the 
air and. holds on to something with the other hand (in the interview they 
were asked to act this out).* All of their explanations of possible re- 
spbnses indicated an understanding of thjs concept, evenS^hen they got 
the question itself wrong. £ 

It seems, in thfs'case, that the youngsters who selected an incorrect 
response did -so for reasons reflecting an accurate construction of the - 

^leaning of this part- of the passage. The range of possible responses per- 
mitted a^Jeast three .semantical ly defensible selections. Of course,. the 
issue of grammatical ity needs to be addressed. In choosing among the al- 
ternatives provided, the readers we observed in this study focussed more 
of their attention *on ^electing a word with # appropriate^meaning than on 
appropriate grammatical ity within the sentence. Students. sometimes ad- ' 
minted their selections di.dn't "sound right" but selected them anyway 
because they had the "right meaning." -Fillmore (Note 2) suggests t&a£ 
young children afid, bilingual individuals may be accustomed to hearing 
constructions with which they are unfamiliar arid therefore, even when 
aware of the "ungrammaticality". of a response, may be less inhibited by 
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, that aspect thafi by imprecise meaning in chops-ing a. response. \ 

Afterthe taped interviews , were transcribed-. and. analyzed, a coding 
sheet was completed for each^tudenfr. At least two raters reviewed the' 
categories to which each response was assigned and also supplied a holistic 
rating of each student's envisionment on a four point scale. Table -1 

9 

compares these envisionment ratings (based on detailed understanding of 
the passages) with performance on the .questions that followed. The 



i Insert Table 1 about here 

results suggest that there is a fairly high association betwee^good en- 
vironment and response accuracy as well as between poor envisionment 
and poor test performance? the trend^s in the right direction. 1 

From thesejmalyses it becomes apparent that individual readers may 
select ^correct test item responses based on interpretations that are 
contextual^' acceptable; these readers' never have the opportunity to 
demonstrate their understanding of the passage. Sometimes' this is because 
the quest'ions-mfss central aspects of the envisionment; in other cases it 
is a function of the idipsyncratic -array of response items. However, those 
readers who are better meaning integrators also tenU to be those who select 
the predetermine correct responses. Findings from our analyses suggest 
that although the test questions themselves do not measure the integration 
of meaning, the"' ability to develop a good envisionment and -the ability 'to 
select a correct response are in some way related. 
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Table 1 



Envisionment 



i. . ' Mean 

■ ,. Number with, % incorrect 

Number of students incorrect Responses responses 



good c 


9 


okay 


-11 


fair 


3 


poor 


3 



1 

5 
2 
2 



6 

23 
33 
67 



'Relationship between incorrect test responses ancUnvis4«nmeRt rating 
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Considerate an'd Inconsiderate Test Items 

A test item is incorrsic!er**e if it makes unwarranted demands on a 
reader, if it creates great cognitive demands without corresponding con- 
ceptual payoff. This is not' -to suggest that all difficult texts are incon- 
siderate'and all easy ones considerate, but that the effort put forth in 

» 

reading a hard text needs to be rewarded. It also suggests that the cog-^ 
nitive demands must be reasonable for the intended audience, the implied 

r 

purpose, and the particular topic. This section will present five aspects 

f *" r- ^ ' 

of inconsiderate tests, aspects that have been apparent in both our *Mdeal. 
reader" text anatysfes and our analyses of real readers reading real test 
items. 'In general, inconsiderate tes't interns create gaps between the reader 
and the. text that are not directly related to how well the reader constructs 
meaning, but are inherent instead in the way the test-maker" composed the 1 
passage or presented the question. Although the five factors are presented 
as discrete categories," in^actual tests any number of them may be inter- 
woven; .one irtconsjderate feature often leads to the emergence of another.' 
Density of Ideas § * 

Test items can be inconsiderate in the number of ideas that ar$ pre- 
sented and the frequency with which these ideas change. The tfrief passages 
used, in test items-^sometimes provide too little elaboration of one concept 

before another concept is introduced. The ideas are too 'short-lived and 

* * 

change too quickly for the reader to de\{|JLop "any adequate envisionment.' 

■Strange Machine (figure 4) is an example of such a text (Note 4).. The ideal 

f 

reader analysis of this item portrays a rapfd succession of new schemata 

* 1 - • ; 

Insert figure 4 about here 



> In 1877 a machine appeared^ which surprised many people. *Can you 
^ guess the name of this»strange new machine? 

t As you spoke into th^ mouthpiece and turned the handle, a tube . 
covered with a thin piece of tin. moved arund. As the tube moved s -«a 
needle pressed deep lines into the tin. ^As you turned th* handle* 
once more, the needle touched .against tfe same lines and played back 
your words. ^ ' ' , 

This. was the first phonograph! t HA* .different from th% Hi-Fi of 
today! ' >-* ^. • 

# — — — — — — ' — — — — — — — 

9 About how many years, ago did the machine ficst appear? 
D «25 2) 50 "3). 100 4) 200 

10 How did people feel' when they saw this new machine? 
5 ) ar W *6)' astonished 

7 ) wom ' ed 5 ' 8) frightened 

11 As the handle was turned, which part o\ the machine moved? 
J) ;tube. 2) needle • 

• 3 ) P re ss 4 ' 4) mouthpiece 

12 As the needle pressed. lines into the tin* it was 
5) giving- a shot 

i 

i 6) • recording a voice 

7) . painting a^picture <V * ^ ,v ' \ 

8) sewing a- piece of material ... 

"' 1 r 
Figure 4: The" Strange Machine 
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introduced without any integration into 'a larger schematic whole,. This 
density is deliberate 'in this passage, inherent in the riddle format and 
postponed answer about the identity of the strange' machine. The reader 
N is asked to, envision a mouthpiece, wfiiqh is associated with a range of 
.possible schemata. Before the schema relevant to this text can be in- 

• 

stantiated, another concept, "handle, i s< 'introduced. The- ideal reader 
•questions" the array of'possible "mean.ings" Q£Co) and forms a rapid succes- - 
sion of hypotheses (HJCo) about the relationships between the parjts of the 
machine. Rather- than finding resolutions te^these hypotheses and questions, 
the density of the text continues' to build- and the "riddle" remains un-* 
solved until the answer is provided .at the end of the passage. • ' 

< While ^e ideal reader notation portrays a series of unresolved 
questions and hypotheses, real. readers make judgments more rapidly even 
when little assistance is provided by the text. In our interviews, some 

* * 

ildren (third -graders) developed art initial envisionment of the "mouth- 

piece" and .clung to ij;, -forcing -each bft of new information into their 

initial envisionment. One reader envisioned a c^rke machine with a "mouthy 

piece" for the coins and a "handle" as the drink .-selector. The "tube" > 

. (introduced next in the*passage) became the rdtuid slot from which -the tin 

coke can was ejected. Others tried to change their envisionments to accom- 

modate the new concepts, but qreated a series of disconnected images instead. 

One reader envisioned a mailbox, then turned- it into a garage mail slot 

(with the slot as mouthpiece), and wherr this no longer worked, envisioned 

a lawnmower with the grass catcher as mouthpiece, the switch as~the handle, 

aTtxt the blades as the tube and needTe. Still .others moved from concept to 

concept without even attempting .to integrate the content. 

* * 
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The density of the. ideas 'introduced without elaboration-is counter- - 
productive to the development of a cohesive envisibnment. Hypotheses 
about the nature of the strange machine (which would integrate the various 
separate rschenfe) are thwarted by another aspect £f the"* text. 15 The ideal ■ 
reader assumes, when the riddle is posed tit the end of the second sentence,' 
that the "strange machine" will be similar enouglvto its 'modern version to 
be gueSsable (Ass.; Genre): In fact, the early version of the machine is 
not at all Tike .the present .version, as the last line of the passage em- 
phasizes. The assumpti6n*of similarity contributes to a series of resolu- 
tions (Res) which invalidate earlier hypotheses, e.g., that the strange 
Machine is a tape recorder, since, the machine recorded and played back 
your words. This particular test item was inconsiderate due to the density 
of ideas, and the lack of resemblance between the early prototype, and pres- 
ent phonograph rendered the passage even more difficult f or *;the reader. 

2. OverVeliance on Assumptions and Hypotheses 

During our investigation we encountered several test items which re- 
quired the ideal reader" to make a large. number of assumptions (Ass), (Co), 
(Tx} and hypotheses (H-*), (Hi), (Co), (Tx) without corroborating textual 
evidence to permit their validation. The-envisionment ,is constructed from 
implications drawn, from the ideal reader's instantiation of unarticulated 
referents and knowledge of^social conventions. While -this can be a powerful 
stylistic device that works welT for texts such as mysteries and\cjothic 
romances, the devic^raquires^space as well as a hospitable setting. When 
many assumptions and urfresolved hypotheses are presentecHn a^text, uncer- 
tainty builds as new suppositions are linked with 1 bid ones, and the reader 
is induced to conjure up a strained envisionment of vague images. 
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, The text'in figure 5 (Note 5), for example*,, may at first glance appear' 

* 

■ 'Insert figure 5 about here 

if ~~ ~~ ."7 — ~ ~ ~ — ~ — — — — — % — o 

simple for. a fifteen year'" old to integrate.- Yet, tobuijd an envistonmeni 
of the meaning of this passage', the reader must rely on many assumptions ' 
and unresolved ^hypotheses. The Ideal reader must make hypotheses abaut 
the ,anteceden€ffor many referents throughout the passage, Jbeginning' i 9 the' 
very first sentence with "a voice," in the second sentence with "the reader" 
and 'iher listener," and so on throughout the text. An assumption must be 

% r 

made, based on social convention?, that the teenage girls -were reading a 
"dirty" book, o< which they feared the mother would ^approve. An assump- 
tion must also' be made 'that the girl and her mother are'taTking across 
"each, other, each concerned with different issues:*- the^girl with' her for- 
bidden bopk, and the mother with her forbidden floor-sitting. The reader 
must also infer that the girl is about fifteen. The^tions (reading the 
book,*hiding it, rising, glancing at the bed) are all -linked by inference. 
This text type is inconsiderate due to the many premature hypotheses the 

reader must make v usual and appropriate inference strategies become tentative 

* * 
and .strained. ^ , " 

3. Imitation Genre , ^ 

There are a variety of genre that consistently 'appeared in the stand- " 
ard^zed reading tests we reviewed: stories, personal accounts, folk tales, 
letters, poems, and expository passages containing information about social 
stddies, science,, or health. Their inclusion seems based on the view that 
a reading comprehension test shoul^ include items from the variety of text 
types that students normally encounter in school. However, our analysis 



"Alice!" called a voic<(. S 
' The effect on the-reader and her listener, both of whom 
were sitting on* the floor, B was instantaneous. Each started and 
, sat rigidly\intent for a moment; then, as the sound of approach- 
ing footsteps was heard, one girl, hastily slipped a. little 
volume under the coverlet 'of the bed, while the other sprang 
to her feet and in a 'hurried, flustered way pretended to be gett 
something out. of a tall wardrobe. . ? , X 

Before the orte who hid the book had time to rise; a woman 

- - ■ >> • • • 

of fifty entered the room and, after a glance, jcr'ied, "Alice! 
How often have I told you not to sit on the "floor?" 

"Very often, Mommy," said Alice, rising' meekly, meantime 
casting a -quick glance at the bed to See how far its smoothness 
had been disturbed. : \ 

. "And still you continue such unbecoming behavior." 
** "Oh*, Mommy, but it is soNa^e, 1 ' cried* the girl. "Didn't 
you like to sit on the floor when you were fifteen?" 



1. Alice's companion was 

A. A girl 

B. Her brother 

C. The family dog, 

D. A doll 

J 



Figure 5 
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of the items disclosed that the passages sometimes violated the conceptual 

or structural patterns typically associated with the' genres they were meant ^ & 

to reflect. In our analyses both of the ideal reader and of real readers,. -'IflSL 

this 'led tQ genre-based hypotheses {-Hyps JSn.^that were later invalidated., 

Although sometimes tlje effects on the envisipnment were mjnor, in other 

cases the'devia^rons from expected genre patterns caused majar interference. 

' ' -For example, a short humorous story generally ends with its punchline. If 

y any, written material follows the punchline, it is likely to elaborate the 

"joke;" it can be read quickly, for embellishment. The reader familiar 

x \ with this genre knows this convention and ; does not expect important infor- 

rna*£ion td follow, However, some test^jrtems place Rey information after 

/iht punchline, or ask questions deali^with what may take place afterwards;* „ 

because this violates a convention of the genre, the reader wfth knbwledge, , 
- * * - 

jk/ of thctt /convention is placed at a ^disadvantage. Figure 6 is an example 
fj* *( * e> , * • ; . 

r of this test and^qtmstiDri type (Note**6)." .Instead of completing the 'actfon** 





Insert figure 6 about herfe , 1 r 

with the punchline, this test items requires the' reader to predict ^hat 
) 'Fritz will do next. This caused som^reiaders^ to doubt their interpretation 

of the punchline and therefore to seT^ct^'response A instead of B. 



The passage in figure 7 (Note 7) is another\xample of a test it&n 
constructed to'imitate a real genre, in this case expository prose, Un-"^ 
like the genV-e the 'passage is imitating, .however, therq,*is no intrpduction 



Insert figure 1 about here 1 



to help the reader make a link from personal knowledge to oew irj/ormation: 
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• Fritz lived in a neighborhood with many interesting 
people'. But Fritz felt out of place, ^e felt that he 
was ordinary. In fact, he felt so ordinary that he thought 
nobody noticed him. Today I'll be different, he thought. 
I'll wear an extra. hat. 

. Then he went for a walk. People smil.ed at him. . But no 
o/ie spoke. At last he asked a neighbor , "Don't you see some- 
thing unusual about me today?" 

"Yes," the neighbor said, "You're wearing three hats 
jnstead of your usual two." * ' ' 



What will Fritz probably do next?' ^ 

A. Ask more people if they think his hat looks different? 

B. Realize that people have always noticed' him 

C. Go to ttoe store, and. bay some mpre hats 

D. Stop going for walks in his neighborhood 

■ * • 

Figure 6 
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The nose cone of a /rocket carries the payload. The 
payload is anything men' wish to send into space. 

• The payload can be 'a satellite. A satellite is a small 
• object that travels around a larger object in space. The 

g|^» rocket travels fast to carry, the satellite away from the 

► earth. 

The payload can be a camera.- It can take pictures of 
. the earth or the moon. The payload may be a man, shooting 
through space. 



ERIC 



16 Which of the following can a space camera be used for? 

5) to take picture of friends 

6) . to measure the temperature 

7) to take pictures of the moon,* 

8) to measure the speed of rockets 

17 . The nose cone carried the '* 

1 ) fuel / * 

2) payload - 

3) . rocket . v 

4) airplane 



V 



*v ^ 8 A small object that travels around a larger one is called a 

5} camera . , 

6) satellite 

7) payload * 

8) nose cone , 

v Figure 7 
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and the passage ends abruptly without a statement tying the passage together. 

al ^conventions of informational genre. Here, the departures from normal 

genre conventions do not mislead, readers, but leave them without needed. 

context for constructing an environment. The reader must begin with some . 

v 

quickly accessible knowledge about nose cones and rockets in order to even 
attempt to develop a cohesive envisionment. There is not a sharing of in- 
formation about rockets from writer to reader, but a simple "listing" of 
key points. The reader who is familiar with the topic will be able to 
reqpgriize these points, elaborate and connect them, and make sense of the 
passage, while the reader who* -is not^alreef^y" familiar with the topic will 
be at a loss. (Here we se§ again how one kind of problem, based on the 
use of an abbreviated genre form creates another, the kind of density of 
ideas discussed earlier.J^J / 

We can see how thi§(can interfere with a developing envisionment ih 
the responses of a third grader interviewed as he read the passage. In the 
first sentence, he knew what rockets looked like (from television) and 
thought he knew'what a nose cone was. He was not certain, but expected 
that it would be elaborated later in the text. He had no schematic links 
between rockets and payload, although he knew from the sentence structure 
that they must have some connection. In the third sentence, he recognized 
the word Satellite and knew it had\something to do with outer space, but 
he had no further knowledge of it. 1 The 'founttj sentence did not help him 
because he couW not ertvision ,what an object in outer space would be like. 

/utyiough this student wis trying to make meaning from the passage 
and had a good idea ofythe information he would' need to know in order to 
comprehend, in this case the text was uncooperative; it did not contain 
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the kinds of "structure and elaboration generally provided in informational 
texts'.' It could make sense only to a reader who already knew' enough about 
' rockets, satellites, and payloads to make the relevant connections among 
them. 

A test 'item is not a .literary work nor an'actual piece of informational. - 
writing. Test writers seem to have other goals in mind than trade book 
authors ; J their purposes and constraints are different. Test items'must 
comply with -a particular reading level, be topically appropriate for specific 
age groups, contain information which can be answered, by specific questions, 
- be inoffensive to all possible groups, and' so on. In an attempt to mimic 
the variety of genre encountered in school and atso to meet these "testing" 
constraints,; test passages might be thought of as "imitation" genre, almost - 
genre unto themselves with conventions of 'their own. It is' this "imitation" 
which renders them inconsiderate: they invoke conventions which they do not' 
follow. ' 

.. 4. Assertions Contrary £0 Readers'' Beliefs ' 
Yet another aspect of inconsiderate text occurs when test items make 
assertions contrary to readers' beliefs without acknowledging differences or 
providing schematic links from reader knowledge^ textual content. Instead, 
, some texts we- analyzed- required the ideal reader to make immediate hypo-, 
theses (Hi, K-») based on an acceptance of the discordant assertion. 

Figure 8 (Note 8) is an example of such a test item, -from a fifth grade 
test. In the: ideal reader notation of the text segment "while attendance 
at baseball games has been falling off," there is a content hypothesis Hyp 
(Co) that falling attendance implies loss in -popularity. There is also a r 

< Insert figure 8 about here 

------- -V^- - - _ 
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Games of great speed and constant motion, such as football, 
basketball, and icq- hockey, haiflg^eeh gaining in popularity, 
while attendance at baseball games has\been- fall ing off. This 
is largely because the game of baseball Ns slower and its action 

\ • " $ 

irregular. That js, the game alternates between great moments w 
of high drama and excitement — such as whe\ the bases are 
' loaded and the batter hits a home ruri — *and Extended periods 
of comparative inactivity, such as when a batteV takes full 
count, two strikes, and three balls, and then is\alked on the 
last pitch. Many ideas are being explored for mak\ng baseball' 
a faster game. One possibil ity., for example, might \e to in- 
crease the distance between 'the pitcher and the batter\ .'thereby 
allowing a split second more for the batter to judge the\pitch 
and connect with the ball. This could produce more hits,\there- 
by making baseball more of "a running rather than a pitching\ame. 



Figure 8 
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text hypothesis Hypl(Tx) that "while" contrasts baseball's falling ' 
attendance with football, basketball, and ice hockey which are'gaining 
^ in popularity. • There are neither ^thematic nor text links elaborating 
this ide,a. The next text segment ("This is Targely because") goes on to 
explore explapations of the loss in popularity. In this segment the ideal 
_ reader makes two hypotheses: Hyp4,(Tx) that "this" refers to baseball's ' 
falling attendance, and Hypi( T x) that "largely" 'suggests more than one 
reason for the falling attendance. ' This is followed by a hypothesis 
Hy P ;(Tx) in the next text segmtnt that baseball is "slower" than other 
.games-. • 

• Real readers interviewed with this text found the assertion about . 
basebarl's loss insularity jarring. Since .children see baseball games 
and baseball players in a wide variety of media contexts, they assume it 
is a very popular game. During an interview! one fifth grader reader 
exhibited well-developed passage environment. However, when choosing the 
response te a question about the decrease in baseball's attendance, he * 
said -angrily , f "I know people are going to come to basebal,! games. I've 
gone to other. people's, houses and they watch. And I've seen the stands " 

4 

just stuffed. They can barely cram another person in." Another reader ' 
knew the individual word meanings of "popularity" (like a movie star, very 
common), "falling off* (not coming to), and "attendance" .(to come to). 
Despite this, his envisionme.nt was poorly integrated. - It wavered between ' 
baseball's gaining andnot gaining in popularity, and therefore baseball's * 
relation to the other games also wavered. Both readers appeared unable, on 
thejr own, ^consistently link the text assertion with their personal* 



beliefs\ab6ut, baseball attendance. This, form of inconsiderate -text re- 
sults when-the author fails to acknowledge the conflicting ideas that 
might arise, and, the text fails to encourage the reader to question per- 
sonally held ideas or beliefs that are contrary, to those assumed in the 
text. 

5. Contextual Discontinuity by Deceptive Simplicity 
The fifth type of inconsiderate text occurs when a concept with which 
a reader is unfamiliar is presented bit-by-bit in a sequence of seemingjy 
familiar details. In the ideal reader analysis, integration^ the details 
leads to an envisionmentof the whole concept. The notation of the Strange 
Machine test item (see Figure 4) is an example). In this passage, the 
strange machine is described bU- by- bit' and the parts can be integrated as 
follows (partial notation): 

f 

As you spoke into the machine and tu rned the handle, 

" : T 1 ' 

1 , • 

e - the clause describes ttie use of the machine in 1877 
Scb(Co) mouthpiece: part ef machine to do with the voice 
Sch(Co) handle: part of machine- helps it to function when turned 
Schematic link - machine, wi th 'mouthpiece and handle,' something 
to ,do with voice communication 

a tube covered with a thin piece of tin moved a round. 
/ : ; 

Q (Co) y/hat is relationship of tube to v mouthpiece anfi handle? 
* Q (Tx) how is the tube moving around? 
H Q (Co) the tutje is also part of the machine ■ % 
e - vh the strange machine, as you speak into the mouthpiece 
and turn fche handle, the* tube turns. 

As the tube moved, a needle pressed deep TThes into'the tin . . . ' 

H(Co) the deep lines' are related to the voice function 

Schematic links - the machine has a needle, mouthpiece, handTe, ■ • 

• and a voice-related function 

H(Co) the lines arepressed to, record a message . 
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~As you turned the handle once more y » 

Ml^Co) the, machine does more than press lines' 
erSj to make the machine work properly the handle must fcfe 
f. turned at least twice ' ' 

t he needle touched against the saipe lines and'plaved back your words . 



Y 



S)ch(Co) voice recording 
Val - a voice recording lVmade 

Val - Q tlre, lines have a nolein* the machine's functioning' 
.Con E - this strange machine allowed people in 1877 to 

record ^teir yoices " . 
H(Co) this was a tape recorder? record player? 

This ttas the first phonograph ! 

Con - the parts ^described are similar enough to the modern day 
, phonograph to call- it the first phonograph - 

How different from the hi fi of today ! , % 

Con - 'both the phonograph and hi fi ,play back sounds and aVe 

thereby related to the^strange machine \ ■ v 

As we have already/seen in^.iscussing^ctensiiy of ideas, this passage 
- leaves the reader with a high level of unintegrated detail. To further * 
complicate' the matter, the vocabulary/is kept deceptively simple, deceptively 
because the concepts introduced have~a wide range of possible referents • 
(e.g., crank, bar, switch, lip for handle) and cak^herefore be attributed 
to a large- number of schemata. Befcause^the conceptual whole is not pro- 
vided by the text, the reader .must cfevelop a& envisionment by relating the 
bits to attributes which are part of an existing textual 1y plausible schema. 
.However, an integration problem occurs if the reader does not already have 
(or is unable to access-) an appropriate schema, if the text concept is so 
remote that it cannot reasonably be related to known schemata, or if the 
text doesn't proyide clues^ to help the reader select among the possible 
schematic choices. • < . 
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This problem was evident iivour interviews when it became apparent 
that although the students were familiar witK tape recorders, the handle', 
tube, and needle did not fit into their notion of a pYototype* , tape re- 
corder. And although they were familiar with ^cord player^, their - 
notion of. a -prototype record player did not include a personal voice- . 
recording function. Further, the third graders we interviewed were not 
familiar with either phonographs or hi fi's although they were quite know!- 

edgeablS about stereos and record players. In the absence of an apparently 

. accommodati 
useful, schematic structure, many studen|$ clung to their .first image,/as 

necessary to new information. For exampt%, one student envisioned (e 3 ) a 

robot with a mouth {mouthpiece), hands' (handles), and a tube with wheels 

to move. The tube was covered with tin "so ydu won f t get a shock." It 

P . . ' ' " 

"^also pressed Tines "so you can see what he says. Je, draws .letters." In 

•this instance these apparently simple text segments evoked schemata which - 

were totally reader-based. This reader did not have the benefit of either 

- * . ' 

text or knowledge. structures .to validate his -hypotheses. Therefore, lack • 

'*« 

of familiarity with the larger whole to which th$ "simple 11 parts relate • 
precludes knowledgeable selection of appropriate* schemata from among the- 
array of those w^ich, in an abstract sense, are potentially possible*. 

Discussion 

* 

This study has examined some of the consequences of test language and 
relationship to comprehension. It has also analyzed test passages in 
terms of the demands test items make on raders. *The two pfeceeding sections 
identify some of the complexities which may severely conftfundT the results 
obtained from standardized tests, complexities which pose processing or 



question-answering, problems unrelated to students' ability to construct 
meantng. . 

, These data^suggest the need ^or caution when considering the pur- ' 
poses for the' administration of tests and what it is that is^fng tested 
If the goal is to make large scale discriminations Between better and 
poorer school achievers, reliance on linguistic and, conceptual "puzzles" 
may suffice. However, if the purpose is to measure Yeading comprehension 
to discriminate among the .range 'of more and less 1 successful users of 
meaning construction strategies, a test specifically designed to exam 
these strategies mast be devised. Our data "indicate that although the f 
reading comprehension tests we examined do discriminate good from poor 
reading, the' strategies require^ fdr* success on. these items do not bear 
a strong relation to the processes involved in constructive meaning-- ■ 

making. Standardised reading tests do not measure the processes involved 

. ♦ • * • _ _ , 

.in the construction o'fSneaning from a text nor do they evaluate an in- 

<-•'/., « 

di.vidual's -ability to manage'those processes. 
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